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SUMMARY 


/ 

This  Is  a  brief  description  of, dynamic  programming# 

.  A 

appear  _in  the  SI  AH  Newsletter  early  in  1956. 
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DYNAMIC  PROGRAMMING 


Richard  Bellman 

It  can  be  said  that  a  field  of  human  endeavor  may  be  called 
a  science  to  the  decree  that  it  Is  quantitative,  and  an  art  to 
the  degree  that  It  Is  qualitative.  Although  decision— making  is 
essentially  an  art  at  the  present  time,  and  will  remain  so  for 
some  time  to  come,  quantitative  elements  are  beginning  to  enter 
to  a  greater  and  greater  degree.  ' 

To  determine  the  soundness  of  our  Judgment,  we  must 
Inevitably  compare  predictions  with  results,  an  ofttimes  expensive 
form  of  experimentation.  A  relativc3y  Inexpensive  way  Is  to 
construct  a  mathematical  model  of  the  underlying  process,  to 
compute  the  quantity  which  corresponds  to  the  outcome  of  the 
original  process,  and.  In  this  way,  to  compare  theory  and 
experiment . 

Since  we  are  usually  Interested  In  maximizing  an  output, 
or  minimizing  a  cost,  we  encounter  In  this  fashion  a  class  of 
variational  problems.  Consider,  for*  instance,  a  situation  in 
which  we  must  make  a  number-  of  decisions  at  each  stage  of  a 
process.  Since  each  decision  corresponds  to  a  choice  of  a 
variable,  we  are  confronted  with  a  multidimensional  pi’oblem  in 
which  the  total  number  of  variables  Is  equal  to  the  number  of 
decisions  to  be  made  at  each  stare  multiplied  by  the  number  of 
stages.  Thus,  for  example,  two  decisions  at  each  stage  of  a 
ten-stage  process  yield  a  twenty-dimensional  variational  problem. 
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Under  certain  simplifying  aasumptlons  where  all  the  functions 
which  arise  may  be  taken  to  be  linear,  there  are  powerful 
methods,  such  as  the  simplex  method  of  George  Dantzlg,  for 
solving  problems  of  this  variety  even  In  situations  Involving 
hundreds  of  variables.  In  addition  there  are  Iterative 
techniques,  such  as  the  "flooding  technique"  of  Alexander 
Boldyreff,  designed  specifically  for  certain  classes  of 
processes . 

If,  however,  the  processes  possess  stochastic  or  nonlinear 
features,  methods  of  the  above  variety  are  usually  helpless, 
and  we  are  left  with  formidable  computational  problems. 

Nonetheless,  quite  often  we  are  rescued  by  the  fact  that 
wo  are  Investigating  a  special  class  of  multidimensional  problems 
arising  from  multxstage  decision  processes,  and  not  completely 
general  multldlmeriSlonal  problems.  Prom  the  principle  of  wishful 
thinking,  we  deduce  that  there  should  be  some  means  of  making 
use  of  this  fact.  Consequently,  In  the  above  example  there 
should  be  a  method  of  formulating  the  mathematical  problem  so 
as  to  reduce  it  to  a  two-dimensional  problem. 

In  order  to  do  this,  we  must  change  our  concept  of  a  solution. 
In  place  of  asking  for  a  method  which  yields  the  entire  sequence 
of  twenty  choices  at  one  time,  as  the  classical  methods  do,  let 
us  ask  for  a  method  which  tells  us  the  optimal  decisions  to  make 
at  each  stage  of  the  process  In  terms  of  the  current  situation. 

This  Is  precisely  the  concept  of  a  "policy''  in  the  physical  world. 
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The  basic  principle  which  enables  us  to  formulate  the 
functional  equations  which  determine  optimal  policies  Is  the 
following ; 

PRINCIPLE  OP  OPTIMALITY:  ^  optimal  policy  has  the  property 
that  whatever  the  Initial  state  and  Initial  decision  are,  the 

I 

remaining  decisions  must  constitute  an  optimal  policy  with 
regard  to  the  state  resulting  from  the  first  decision. 

A  further  advantage  of  the  above  approach,  apart  from  the 
drastic  reduction  In  dimensionality,  Is  that  It  Is  Ideally 
suited  to  processes  Involving  random  effects.  Furthermore,  It 
provides  a  new  approach  to  the  calculus  of  variations  which 
enables  us  to  resolve  by  numerical  means,  and  occasionally 
analytic  means,  a  variety  of  problems  arising  In  appllcptlons 
which  cannot  easily  be  treated  by  other  means. 

Applications  of  the  theory  of  dynamic  programming  have 
been  made  to  flnite-dlmenslonal  maximization  problems,  the 
calculus  of  variations.  Industrial  production  and  allocation 
problems,  scheduling  theory,  logistics,  control  processes, 
multistage  games,  and  to  a  variety  of  military  problems.  A 
bibliography  of  papers  up  to  195^  may  be  found  In 

Bellman,  R.,  "The  Theory  of  Dynamic  Programming,"  Bull.  Amer. 


Math.  Soc.,  Vol.  60,  195^,  pp.  503-516. 


